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RESUME 


La  poursuite  d'une  cible  par  la  correlation  d' images  consiste  b 
correier  une  image  etalon  de  la  cible  avec  une  scfene  prise  sur  le  vif. 
Le  grossissement  et  les  rotations  de  la  cible  diminuent  considerablement 
les  performances  de  l'algorithme  de  correlation.  Le  present  rapport 
decrit  deux  algorithmes  qui  permettent  b la  t§te  chercheuse  de  compenser 
l'effet  du  grossissement  et  des  rotations  b l'aide  d'une  lentille  zoom 
et  d'un  systfeme  de  balanciers.  Un  premier  algorithme,  applicable  b des 
cibles  fixes,  evalue  l'angle  de  rotation  par  rapport  b la  ligne  de  visee 
(roulis).  Ouant  au  second,  il  determine  l'angle  de  rotation  selon  les 
trois  axes,  ce  qui  est  necessaire  pour  une  cible  en  mouvement.  Une 
simulation  realisee  b partir  de  photographies  et  d' images  artif icielles 
demontre  l'efficacite  de  ces  deux  algorithmes.  (NC) 


ABSTRACT 

\ 

Target  tracking  by  area  correlation  involves  matching  a stored 
reference  image  of  the  target  to  live  search  scenery.  The  performance 
of  the  area  correlator  can  be  severely  degraded  by  target  magnification 
and  three-axis  rotation.  This  report  presents  2 algorithms  that 
estimate  magnification  and  rotation  angles,  and  allow  the  tracker  to 
take  compensating  action  via  zoom  lens  and  roll  gimbal.  The  first 
algorithm,  applicable  to  fixed  targets,  estimates  rotation  about  the 
line  of  sight  only  (roll  angle).  The  second  algorithm  estimates 
rotation  in  3 axes,  as  required  for  maneuvering  targets.  The 
presentation  of  both  algorithms  is  supported  by  simulation  arsing 


photographs  and  artificial  images.  (U) 
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DEFINITION  OF  SYMBOLS 


Ax  = horizontal  translation  (along  x-axis) 

Ay  = vertical  translation  (along  y-axis) 

M = magnification  ratio 

a = angle  of  rotation  about  x-axis 

e = angle  of  rotation  about  y-axis 

Y = angle  of  rotation  about  z-axis  (line  of  sight) 

i.e.  roll  angle 

r = total  roll  angle  between  sensor 

axes  and  pitch  and  yaw  axes 
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1.0  I PRODUCTION 

Area  correlation  tracking  is  an  effective  technique  for  tracking 
targets  that  have  neither  prominent  features  nor  high  contrast  with  the 
background.  Essentially,  this  method  involves  matching  a stored 
reference  image  of  the  target  to  real-time  scene  data  acquired  by  an 
imaging  sensor.  The  image-matching  is  accomplished  by  superposing  the 
reference  image  over  the  search  scene  at  all  possible  positions,  and 
computing  the  corresponding  values  of  some  kind  of  image  similarity 
measure.  The  position  within  the  search  scene  corresponding  to  maximum 
image  similarity  is  taken  to  be  the  target's  location.  A basic 
configuration  of  an  area  correlation  tracking  system  is  shown  in  Fig.  1. 

In  a tracking  application,  the  image-matching  algorithm 
encounters  problems  of  target  magnification  and  rotation.  In  the  field 
of  missile  guidance,  these  problems  have  been  circumvented  by  a 


periodical  updating  of  the  reference  target  image.  That  is,  new 
reference  images  of  the  target  are  repeatedly  selected  from  the  incoming 
live  search  scenes.  This  procedure,  however,  is  subject  to  errors  in 
computed  target  position.  These  errors  accumulate  and  can  cause  the 
tracker  to  wander  away  from  the  target. 


The  work  presented  in  this  report  addresses  the  problem  directly 
by  deriving  algorithms  that  estimate  relative  target  magnification  and 
rotation  as  well  as  position.  These  estimates  are  then  used  in 
controlling  compensating  devices  (zoom  lens,  gimbals,  etc.)  to  maintain 
the  target  at  the  center  of  the  field  of  view  with  a constant  scale  and 
orientation.  This  allows  continued  tracking  using  the  original 
reference  target  image  without  updating. 
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The  estimates  of  magnification,  rotation,  and  position  are 
obtained  by  estimating  the  coordinate  transformation  between  a grid  of 
points  in  the  reference  image  and  the  corresponding  grid  in  the  search 
scene.  Hie  search  grid  that  corresponds  to  the  reference  grid  is  formed 
by  applying  the  image-matching  algorithm  to  relatively  small  image 
subsections  at  each  point  of  the  reference  grid. 

The  estimation  problem  can  be  two-  or  three-dimensional, 
depending  on  the  exclusion  or  inclusion  respectively  of  target  rotation 
about  perpendiculars  to  the  line  of  sight.  That  is,  if  there  is  no 
rotation  about  perpendiculars  to  the  line  of  sight,  then  the  grid 
coordinate  transformation  can  be  fully  described  in  two  dimensions.  If 
there  is  rotation  about  perpendiculars  to  the  line  of  sight,  then  three 
dimensions  are  required.  Both  cases  are  treated  in  this  report.  In 
general,  fixed  targets  require  only  a two-dimensional  grid  analysis, 
while  maneuvering  targets  require  three  dimensions. 

A major  difficulty  in  the  three-dimensional  case  consists  in 
compensating  for  rotation  about  perpendiculars  to  the  line  of  sight. 
While  effects  such  as  target  translation,  magnification  and  roll  can  be 
eliminated  via  gimbals  and  a zoom  lens,  there  is  no  corresponding 
corrective  action  for  the  case  of  a target  turning  away  from  the  sensor. 
Consequently,  reference  image  updating  will  be  required  as  the  target 
rotates  about  perpendiculars  to  the  line  of  sight.  Target  image  scale 
and  roll  orientation  are  maintained  constant. 


The  tracking  system  acquires  an  important  capability  when  it  is 
equipped  with  the  three-dimensional  grid  analysis  algorithm.  Without 
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it,  the  system  tracks  only  an  image  which  can  rotate  out  of  view.  With 
3 dimensions,  on  the  other  hand,  it  will  be  shown  that,  within 
limitations,  the  system  tracks  the  target  as  a true  object  rather  than 
as  an  image. 

This  work  was  performed  at  DREV  between  March  and  September  1977 
under  PCN  21J03  "Lnaging  Seekers". 

2.0  AREA  CORRELATION 

Image-matching  by  area  correlation  consists  of  scanning  a 
reference  image  over  a search  image  and  recording  the  location  of  the 
maximum  computed  value  of  a similarity  function.  This  function  can  be 
the  two-dimensional  cross-correlation,  as  implied  by  the  term  "area 
correlation",  but  this  has  been  generalized  to  include  the  use  of  any 
meaningful  similarity  function  or  measure.  For  the  reader’s  reference, 

3 algorithms  corresponding  to  3 similarity  measures  will  be  briefly 
descr ibed. 

2.1  Cross-Correlation  Algorithm 

The  reference  and  search  images  are  digitally  represented  as 
two-dimensional  arrays  of  integers  that  correspond  to  image  intensity. 

Let  these  images  be  denoted  by  matrices  rt  and  S (reference  window  and 
search  scene  respectively).  Then  the  simple  cross-correlation  function 
(Refs.  1-3)  is  given  by 
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R(u,v)  = ZZ  suv(i» j)  W(i,j)  [1] 

where 

Suv(i,j)  = S(u+i,v+j) 

As  a measure  of  image  similarity,  this  function  can  easily  be  tricked  in 
other  areas  of  the  search  scene  that  correlate  highly  because  of  their 
br ightness. 


i 


These  false  correlations,  which  result  from  nonuniform  image 
contrast  and  dc  level,  can  be  eliminated  by  incorporating  normalization 
in  the  correlation  function.  Hence  the  normalized  cross-correlation 
function  is 


R(u,v)  = ZZ 

ij 


111) 

1J  \ uv  / \ ' 


[2] 


— — 2 2 

w*iere  Suv'  W are  an^  og  , aw  are  local  variances.  At 

every  spatial  shift  (u,v) , before  multiplying  and  summing  (correlating) , 
this  function  normalizes  the  imaqes  to  identical  mean  values  (dc  levels) 
of  zero  and  identical  variances  (contrasts)  of  unity.  It  can  be  readily 
shown  that  maximizing  the  normalized  cross-correlation  function  [2]  is 
equivalent  to  minimizing  the  normalized  mean  sauare  error 

[3] 


«c.„  ■ « [ " ) 
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aicnougn  normalize:  cross-correlation  is  a robust  and  reliable 
image-matching  algorithm,  it  is  very  time-consuming.  In  this  report,  it 
is  used  primarily  with  small  reference  and  search  a^eas  after  a coarse 
search  has  been  done  by  a faster  algorithm. 


2.2  Mean  Absolute  Difference  Algorithm 


The  mean  absolute  difference  (MAD)  algorithm  (Refs.  4-5)  is  a 
somewhat  faster  algorithm  that  reDlaces  the  product  operation  of 
cross-correlation  by  the  absolute  value  of  the  difference.  It  is  given 


ana  is  a function  to  be  minimized.  For  most  arithmetic  processors,  the 
MAD  algorithm  has  a notable  speed  advantage  over  cross-correlation 
because  the  combined  subtraction  and  absolute-value  operation  is  less 
time-consuming  than  multiplication.  This  algorithm  is,  however,  not  as 
robust  in  the  presence  of  significant  image  noise  and  distortion  (Ref. 
4).  A normalized  version  of  the  MAD  algorithm  is 


UNCLASSIFIED 

7 


2.3  Sequential  Similarity  Detection  Algorithm 


The  sequential  similarity  deletion  algorithm  (SSDA)  is  a 
high-speed  image-matching  algorithm  presented  in  Ref.  6 and  analyzed  in 
Ref.  7.  The  basis  of  the  SSDA  is  the  observation  that  a great  deal  less 
computation  is  required  to  determine  image  dissimilarity  than  is 
required  to  determine  image  similarity. 

The  image-comparison  criterion  of  the  SSDA  is  the  same  as  that  of 
the  MAD  algorithm:  the  sum  of  the  absolute  values  of  the  differences 
between  picture  elements.  In  the  SSDA,  however,  this  summation  is  not 
necessarily  performed  to  completion.  The  picture  elements  are  taken  in 
random  nonrepeating  order  (so  as  to  maximize  new  information  at  each 
step)  and  at  every  step  of  the  summation,  the  sum  is  subjected  to  a 
threshold.  When  the  threshold  is  exceeded,  the  comparison  is  halted  and 
the  number  of  steps  required  to  cross  the  threshold  is  recorded  as  the 
function  of  image  similarity.  This  function  is  small  in  regions  of 
image  dissimilarity  where  picture  element  differences  are  large  and 
their  summation  crosses  the  threshold  at  an  early  stage.  Hiis  also 
means  that  very  little  computation  is  expended  in  areas  of  image 
dissimilarity.  Since  the  vast  majority  of  candidate  match-points 
correspond  to  image  dissimilarity,  there  is  a drastic  overall  reduction 
in  computation. 

Figure  2 illustrates  the  typical  operation  of  the  SSDA.  Note 
that  the  threshold  is  not  a constant  throughout  the  summation  of  picture 
element  differences.  This  is  so  that  comparison  of  dissimilar  images 
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Number  of  Steps 


FIGURE  2 - Typical  operation  of  the  SSDA.  When  images  are  dissimilar 
(D) , cumulative  error  grows  quickly,  giving  low  value  of 
similarity  function.  With  similar  images  (S),  error  grows 
slowly  and  similarity  function  is  high. 
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can  terminate  as  early  as  possible,  and  comparison  of  similar  images  can 
proceed  fully.  Hie  curve  D is  a summation  of  picture  element 
differences  of  dissimilar  images.  It  is  seen  that  the  threshold  is 
crossed  early,  giving  a low  value  (R^)  of  similarity.  The  curve  S 
corresponds  to  the  case  of  similar  images  and  crosses  the  threshold  at  a 
late  stage,  yielding  a high  value  (R9)  of  similarity. 

The  performance  of  the  3SDA  is  highly  dependent  on  the  threshold 
sequence  used.  A low  threshold  leads  to  a sharp  peak  in  the  similarity 
function  and  a greatly  reduced  computational  effort.  But  a low 
threshold  also  increases  the  probability  of  terminating  summation 
prematurely  and,  thereby,  missing  the  correct  image  match-point.  The 
determination  of  an  optimum  threshold  is  a difficult  problem  because  of 
the  SSDA's  high  degree  of  nonlinearity.  Reference  6 presents  two 
heur istically  derived  families  of  threshold  sequences  that  work  well 
although  they  are  not  designed  to  be  optimal  in  any  way.  Reference  7 
gives  an  analysis  that  predicts  the  performance  of  the  SSDA  for  these  or 
any  other  threshold  sequences. 

2.4  Magnification  and  Rotation 

Since  the  image-matching  is  essentially  a search  for  a best 
superposition  of  images,  it  should  be  evident  that  this  task  is  impeded 
by  differences  in  magnification  and  orientation  angle  between  the  images 
being  compared.  In  addition,  the  sensitivity  to  magnification  and 
rotation  increases  with  increasing  detail  present  in  the  images.  In 
other  words,  the  overlap  of  images  of  different  scale  or  orientation  is 
weakest  in  areas  containing  fine  detail.  This  is  illustrated  in  Fig.  3 


j 
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where  a comb  structure  is  superposed  with  magnified  and  rotated 
versions.  It  is  apparent  that  the  fine  detail  of  the  teeth  retains  less 
correlation  under  geometric  distortion  than  the  broad  feature  of  the 
handle. 


The  important  point  to  note  here,  however,  is  that  although 
entire  comb  pictures  may  not  overlap  well,  small  segments  can  be  made  to 
overlap  relatively  well  individually.  In  Figures  3(b)  and  (c) , for 
example,  the  4 teeth  in  the  center  of  the  comb  overlap  to  a good  extent 
with  no  translation  required.  The  3 teeth  at  each  end  of  the  comb  can 
also  be  made  to  overlap  if  translations  are  applied.  Thus,  the  result, 
rather  than  being  a single  position  for  the  second  comb,  is  a set  of 
translations  describing  a rubber-sheet  stretching  effect.  From  this  set 
of  translations  of  individual  segments,  it  is  possible  to  estimate  the 
position,  the  magnification  and  the  rotation  angle  of  the  second  comb. 
This  is  the  essence  of  the  technique  presented  in  this  report  for 
estimating  position,  magnification,  and  rotation. 


3.0  THE  TWO-DIMENSIONAL  COMPENSATION  PR0BI£M 


In  the  grid  analysis  technique,  the  geometrical  relationship  of  a 
pair  of  images  is  given  by  the  geometrical  relationship  of  a 
representative  subset  of  corresponding  points  in  the  images.  Thus,  a 
grid  of  points  is  defined  in  the  reference  image,  and  the  corresponding 
grid  in  the  search  scene  is  located  by  image-matching.  The  geometrical 
relationship  between  the  two  grids  is  determined  by  estimating  a 
coordinate  transformation  between  them.  When  tt..*re  is  no  target 
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rotation  about  perpendiculars  to  the  line  of  sight,  the  problem  is 
two-dimensional  in  that  only  two  coordinates  are  required  to  describe 
the  grids  and  their  relationship. 

The  mathematical  derivation  of  the  estimation  of  the  coordinate 
transformation  between  a pair  of  grids  is  given,  followed  by  the  results 
of  a simulation  test  on  still  photographs.  Extension  of  the  problem  to 
three  dimensions  is  treated  in  Section  4.0. 

3.1  Estimation  of  a Two-Dimensional  Coordinate  Transformation 


A 25-point  reference  grid  and  a 25-point  search  grid  are  shown  in 
Fig.  4.  The  search  grid  corresponds  to  a target  that  is  translated, 
magnified,  and  rotated  relative  to  its  reference  attitude.  The  points 
of  the  search  grid  are  found  by  image-matching  and,  therefore,  are 
subject  to  errors  in  their  location. 

The  geometrical  relationship  between  pairs  of  corresponding 
points  can  be  described  in  two  dimensions  by  a coordinate  transformation 
of  the  form 


cosy 

-siny 


siny 

cosy 


[6] 


where  (x,y)  are  coordinates  of  a reference  grid-point,  (u,v)  are 
coordinates  of  a search  grid-point,  M is  the  magnification  of  the  9earch 
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image  relative  to  the  reference  image,  y is  the  clockwise  roll  angle  of 
the  search  image  with  respect  to  the  reference  image,  and  ax  and  Ay  are 
the  horizontal  and  vertical  translations  of  the  search  image  relative  to 
the  reference  image  expressed  in  the  reference  frame. 


The  reason  for  obtaining  the  translations  in  the  reference  frame 
rather  than  in  the  search  frame  is  the  intention  that  a zoom  lens  and 
inner  roll  gimbal  bring  the  live  search  image  to  the  original  reference 
scale  and  roll  angle.  Given  this  correction,  the  necessary  centering 
translations  to  be  applied  to  the  sensor  (via  outer  tracker  pitch  and 
yaw  gimbal  adjustments)  are  those  expressed  in  the  reference  frame  of 
coordinates.  As  a result  of  roll  corrections,  the  sensor  axes  and  the 
pitch  and  yaw  gimbal  axes  are  not  generally  coincident.  The  (Ax,  Ay) 
signals  must  therefore  be  prerotated  by  the  total  roll  angle  between  the 
sensor  axes  and  the  gimbal  axes.  Thus,  the  signals  feeding  the  gintoals 
are 


cos  r 
-sin  T 


sin  T 
cos  r 


where  r is  the  total  roll  angle.  The  ax'  signal  is  sent  to  the  yaw 
gimbal  control,  and  the  Ay1  signal  is  sent  to  the  Ditch  gimbal  control. 

A least  squares  estimate  of  the  coordinate  transformation  is 
given  by  the  values  of  M,  y , ax  and  Ay  that  minimize  the  mean  square 


- 

; 

I 

i 

' 


tijiwi  - — t=.  c rr/f’  zz 


error  of  [6]  for  the  ensemble  of  grid  points.  Let  the  grid  data  be 
arranged  as  N-vectors  x,  u,  v.  Then  the  mean  square  error  is 


MSE  = I I u - M(x-Ax)cosy  - M(^-Ay)siny  | |2 

[7] 

1 2 
+ II  v_  + M(£-Ax)siny  - M(^-Ay)cosy  |j 

l 

2 

where  ||  ||  indicates  vector  magnitude  squared.  Eauating  partial 

derivatives  of  [7]  to  zero  yields,  after  somewhat  lengthy  algebra,  the 
following  estimates: 


\ 
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where  x,  y,  u,  v are  mean  values  of  the  elements  of  vectors  x,  u,  v, 
and  Cux  is  the  covariance  of  the  elements  of  vectors  u and  x;  that  is, 
Cux  = (u-u)  (x-x)  etc. 

What  will  be  referred  to  as  the  2-D  tracking  algorithm  can  then 
be  described  as  repetition  of  the  following  sequence: 

1)  Acquire  the  search  scene 

2)  Determine  the  search  grid  corresponding  to  the  reference  qrid 
by  image-matching. 

3)  Perform  a two-dimensional  grid  analysis  to  estimate  Ax,  Ay,  y 
and  M. 

4)  Send  ax'.  Ay'  to  sensor  pitch  and  yaw  control. 

5)  Send  M to  zoom  lens  control. 

6)  Send  y to  sensor  roll  control. 

Although  tv*>  point-pairs  are  sufficient  to  define  a 
two-dimensional  coordinate  transformation,  errors  in  the  search 
grid-point  locations  dictate  that  considerably  more  than  two  grid-points 
be  used.  The  use  of  a large  number  of  grid-points  improves  the  estimate 
of  the  coordinate  transformation  by  increasing  the  error-averaging 
effect.  As  an  example,  if  N grid-points  are  arranged  on  a circle 
(rather  than  on  a square  array)  and  if  the  search  grid  errors  are 
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zero-mean,  it  can  be  shown  that  the  error  in  the  magnification  estimate 
is  inversely  proportional  to  /N. 

In  obtaining  the  search  grid  point  locations,  it  is  not  necessary 
to  perform  a wide  search  for  each  subreference.  Once  the  general 
location  of  the  target  is  determined  from  a single  wide  search  with  a 
large  reference  (32  x 32,  for  example) , the  subreference  locations  may 
be  found  with  relatively  small  searches  in  the  vicinity  of  the  expected 
locations.  Hie  size  of  these  subsearches  is  determined  by  the  maximum 
expected  relative  magnification  ratio  and  roll  angle.  For  reasonable 
magnification  and  roll  rates,  these  subsearches  are  considerably  smaller 
than  the  search  for  overall  target  position.  Hence,  computationally, 
the  grid  analysis  algorithm  does  not  add  great  expense  to  the  basic  area 
correlation  algorithm. 

3.2  Simulation  Test 


The  two-dimensional  grid  analysis  technique  was  applied  to  an 
aerial  infrared  photograph  of  a highway  scene.  The  128  x 128  reference 
image  of  the  highway  is  shown  in  Fig.  5(a)  with  the  target  area 
segmented  into  a 4 x 4 array  of  sixteen  16  x 16  subreferences.  The 
center  points  of  these  subreferences  define  a 4 x 4 reference  grid  of  16 
points.  The  128  x 128  search  scene  shown  in  Fig.  5(b)  is  a 
computer -processed  version  of  the  reference  image.  The  operations 
performed  on  the  reference  image  to  produce  the  search  image  are  a 
horizontal  translation  of  2 picture  elements,  a vertical  translation  of 
5 picture  elements,  a magnification  by  a factor  of  1.1,  and  a clockwise 
roll  rotation  of  5 degrees. 
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(a)  Reference  image  (b)  Search  image 


FIGURE  5 - Aerial  infrared  highway  imagery 


Each  point  of  the  search  grid  was  determined  by  matching  a 
subreference  image  to  an  area  of  the  search  scene.  The  image-matching 
algorithm  used  for  this  purpose  was  the  SSDA.  Fractional  coordinate 
values  for  each  point  of  the  search  grid  were  computed  as  the  centroid 
of  a 5 x 5 region  about  the  peak  of  the  SSDA  similarity  function. 

Ttoe  reference  and  search  grids  for  this  test  are  shown  in  Fig.  6. 
Note  that  two  points  of  the  search  grid  (upper  right)  are  missing 
because  the  associated  subreference  images  could  not  be  uniquely  found 
in  the  search  scene.  In  the  case  of  the  extreme  upper-right-corner 
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Y 


FIGURE  6 - Highway  reference  and  search  grids.  The  dots  represent 
the  locations  of  the  center-points  of  the  reference  and 
search  subimages. 
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subimage  (Fig.  5(a)),  there  is  very  little  content  to  distinguish  that 
particular  area  from  its  surroundings.  The  resulting  similarity 
function  was  therefore  too  flat  to  allow  any  kind  of  peak  determination. 
Similarly,  in  the  case  of  the  subimage  immediately  to  the  left  of  this, 
the  nearly  vertical  section  of  road  is  virtually  indistinguishable  from 
vertically  adjacent  sections  of  road.  Here,  the  similarity  function  was 
a uniform  nearly  vertical  ridge  from  which  no  y-coordinate  could  be 
established.  Cases  such  as  these  can  be  weeded  out  prior  to  correlation 
with  live  search  imagery  by  testing  subreference  autocorrelation.  If  a 
subreference  fails  to  be  distinct  within  the  reference  image  itself, 
then  that  subreference  and  the  corresponding  grid-point  are  not  to  be 
used. 


The  remaining  14  pairs  of  grid  coordinates  were  entered  into  eqs. 
8.1  to  8.3.  Ihe  exact  coordinate  transformation  parameters  were 

Ax  = 2 
Ay  = 5 
M = 1.1 

Y = 5° 

The  estimated  parameters  given  eqs.  8.1  to  8.3  were 

Ax  = 2.320 
Ay  = 5.128 
M = 1.093 

Y = 5.276° 


These  estimates  are  used  to  direct  a zoom  lens  and  a roll  gimbal 
to  bring  the  search  scene  back  to  the  scale  and  roll  attitude  of  the 
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reference  image.  The  estimates  reduce 
sufficiently  to  allow  continued  tracking  with 


the  orientation  errors 
the  original  reference 


image. 


A discussion  of  the  practical  aspects  of  this  technique  applied 
to  missile  guidance  will  be  left  to  Section  5.0.  At  this  point,  the 
problem  will  be  extended  to  three  dimensions  to  include  consideration  of 
target  rotation  about  perpendiculars  to  the  line  of  sight. 


4.0  THE  THREE-DIMENSIONAL  COMPENSATION  PROBLEM 


To  determine  a general  description  of  a target's  attitude  in 
space  (including  rotation  angles  in  three  axes) , three-dimensional 
information  about  the  grid-points  is  required.  The  third  coordinate 
represents  relative  range  (depth)  along  the  line-of-sight  (z-axis) . An 
image  of  a target  is  a tv*3-dimensional  representation,  however,  that 
does  not  give  direct  range  information.  To  avoid  the  complication  of 
integrating  a range-finding  instrument  into  the  tracking  system,  the 
algorithm  presented  in  this  section  estimates  the  range  information  as 
well  as  the  coordinate  transformation  parameters. 

4.1  Estimation  of  a Three-Dimensional  Coordinate  Transformation 


A coordinate  transformation  in  three  dimensions  is  given  by 
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"u  ' 

MR  R R 

x-Ax" 

V 

Y 6 a 

y-Ay 

_ w _ 

z 

where  (x,y,z)  are  reference  coordinates,  (u,v,w)  are  search  coordinates, 
M is  the  magnification  factor.  Ax  and  Ay  are  translations,  and  R , R 

a 0 

and  are  matrices  of  rotation  about  the  x,  v and  z axes  resoectively. 
These  matrices  are 


1 0 0 
0 cosa  sina 
0 -sina  cosa 


[10.1] 


R = 
Y 


‘cose 

0 

-sinB  " 

0 

1 

0 

sing 

0 

cosB  _ 

cosy 

siny 

°1 

-siny 

cosy 

o 

[10.2] 


[10.3] 


The  geometry  of  rotation  is  illustrated  in  Fig.  7.  It  is  required  to 
estimate  Ax,  Ay,  M,  a,  6 and  y given  a reference  grid  and  a search  grid 
described  only  by  sets  of  (x,y)  and  (u,v) . Range  coordinates  z and  w 
are  unknown. 

The  equations  relating  the  qiven  (x,y)  and  (u,v)  grid  data  can  be 
rewritten  as 
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r,n 

rll 

r12 

r13 

' x-Ax  ' 

v = M 

y-Ay 

LVJ 

_ r21 

r22 

r23  _ 

L 

vrtiere  r.  . are  elements  of  the 

first 

two 

rows 

of  the 

[11] 


R Re  R : 
y B a 


= cosg  cosy 

= sina  sing  cosy  + cosa  siny 
= -cosa  sing  cosy  + sina  siny 
r21  = -cosg  siny 

r22  = -sina  sing  siny  + cosa  cosy 
r2j  = cosa  sing  siny  + sina  cosy 


[12] 


As  in  the  two-dimensional  problem,  it  is  sought  to  minimize  the 
mean  square  equation  error 


MSE 


I |u  - Mrn  (x  - Ax)  - Mr12  - Ay)  - Mr13  zj 


113] 


_ | |v  - Mr21  (x  - Ax)  - Mt22  (£  - Ay)  - Mr2,  z_ 


22 


23 


v*iere  the  ensemble  of  coordinate  data  are  again  arranged  as  vectors. 
Note  that  it  is  necessary  to  determine  the  unknown  range  vector  z,  as 
well  as  the  parameters  ax,  Ay,  M,  a,  fi  and  y. 
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Equating  partial  derivatives  to  zero  leads  to  a set  of 
transcendental  algebraic  equations  to  which  a complete  analytical 
solution  was  not,  in  fact,  obtained.  Instead,  an  algorithm  of 
successive  approximation  was  derived.  It  consists  of  a sequential 
application  of  analytical  solutions  to  a set  of  subestimation 
(subminimization)  problems. 

Ihe  first  iteration  of  the  successive  aporox imation  algorithm 
consists  of  the  following  sequence  of  steps: 

1)  Estimate  Ax,  Ay,  M,  y 
assuming  a = 6 = z = 0. 

2)  Estimate  Ax,  Ay,  M,  B,  z 
given  y from  step  1 
assuming  a = 0. 

3)  Estimate  ax.  Ay,  M,  a,  z 
given  y from  step  1 
assuming  B = 0. 

4)  If  MSE  of  step  2 < MSE  of  step  3 
then  estimate  a.  Ay 

given  ax,  M,  y,  8,  z of  step  2. 

If  MSE  of  step  3 < MSE  of  step  2 

then  estimate  B,  ax 

given  Ay,  M,  y,  a,  z of  step  3. 


UNCLASSIFIED 

2r-> 


5)  Estimate  ax,  ty,  z 

given  t,  u,  8,  H of  step  4. 

6)  Estimate  ax.  Ay,  M,  y 
given  a,  8,  z of  step  5. 

Succeeding  iterations  consist  of  repetitions  of  steps  4,  5 and  6 with 
step  4 modified  as  follows: 

4)  Estimate  a.  Ay 

given  ax,  M,  y,  8,  z of  steo  6, 
and 

estimate  B,  ax 

given  Ay,  M,  y,  a,  z of  step  6. 


Step  1 is  equivalent  to  the  two-dimensional  problem  and  requires 
no  elaboration.  The  solution  is  given  in  Section  3.1  by  equations  8.1 
to  8.3.  As  will  be  seen  later,  this  step  generates  a good  estimate  for 
roll  angle  y in  spite  of  the  fact  that  the  true  transformation  is 
three-dimensional . 


Step  2 accepts  the  y estimate  and  re-estimates  the  remaining 
parameters,  including  8 and  z,  with  a still  assumed  to  be  zero.  With 


data  replaced  by 


expressed  as 


cosy  siny"l  * u"| 

-siny  cosy  I vl 


the  transformation  can  be 
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The  solution  is  not  completely  determined  without  specifying  range 
statistics  z and  C as  well  as  the  sign  of  the  angle  B.  In  other 

ZZ 

words,  there  are  an  infinite  number  of  solutions  that  will  optimally 

relate  the  reference  grid  to  the  search  grid.  For  example,  if  C is 

zz 

large,  B will  be  small. 

This  nonuniqueness  is  a reflection  of  the  loss  of 
three-dimensional  information  in  the  process  of  imaging.  Without 
additional  external  information,  an  image  provides  no  depth  information. 
Oice  the  angle  between  two  views  of  an  object  is  known,  two  images  can 
provide  three-dimensional  information.  This,  in  fact,  is  the  basis  of 
stereometric  depth  perception  (Refs.  8-9).  In  the  tracking  problem, 
however,  the  angle  between  two  views  is  not  known  and  hence  unique 
determination  of  range  is  not  possible.  Even  by  using  additional  views 
and  simultaneously  matching  multiple  search  grids  (consecutive  in  time) 
to  the  reference  grid  one  would  not  uniquely  determine  range  without 
using  angular  information. 


In  application,  values  of  z and  C are  therefore  assigned  prior 
to  the  tracking  mission.  The  sign  of  B can  be  determined  on-line, 

however,  by  requiring  that  the  computed  range  coordinates  of  the 

reference  grid  represent  a convex  surface.  This  is  based  on  the 

j 

assumption  that  the  target  is  a reasonably  shaped  object  whose  front 

j 

surface  as  seen  in  the  reference  image  is  essentially  convex  rather  than 
concave.  If  the  sign  of  B is  wrong,  the  resulting  range  estimates 

describe  a reflection  of  the  true  surface;  that  is,  a convex  surface 


becomes  concave  (and  vice  versa) . The  test  for  convexity  consists  in 
computing  a coefficient  of  fit  of  the  estimated  range  data  to  a 
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paraboloid.  The  sign  of  3 is  taken  to  be  the  one  that  Takes  this 
convexity  coefficient  oositive. 

Step  3 is  a parallel  of  step  2 but  this  time  a is  estimated  and  6 
is  assumed  to  be  zero.  In  the  meantime,  the  results  of  steo  2 are 
stored  for  later  use.  The  transformation  then  is 


M 


1 0 
0 cosa 


-• 

0 

x-Ax 

y-Ay 

sina 

2 

_ 

[16] 


where,  as  in  step  2,  the  (u,v)  coordinates  have  been  prerotated  by  a 
roll  angle  of  y.  The  estimates  are 


Ay  = y - 


V 


♦ 2 tana 


[17.5] 


Mcosa 


Hie  same  values  of  z and  C 2z  as  in  steo  2 are  used,  of  course,  and  the 
sign  of  a is  again  determined  by  a convexity  test  on  z. 

Step  4 compares  the  parallel  estimates  of  steps  2 and  3 and 
accepts  whichever  yields  the  smaller  mean  square  error.  If  the  result 
of  step  2 is  chosen  then  step  4 will  estimate  a and  av  using  estimates 
of  ax,  M,  6,  y , and  z at  the  end  of  step  2.  The  procedure  is  to  first 
compute  w and  prerotate  (u,v,w)  by  anqles  y and  6.  The  oroblem  is  then 
simplified  bo  that  of  determining  the  transformation 


v = M [(y-Ay)cosa  ♦ z sina] 


[18] 


Hie  parameter  Ay  is  given  by 


_ v _ 

Ay  = y - + z tana 

Mcosa 


[19.1] 


and  a is  given  implicitly  by 
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C cosa  - C 

sina  + 

vz  vy 

+ M C 

r • 2 

(sin  a 

yz 

M(C  - C )sina  cosa 
yy  zz 

- cos^a)  = 0 


[19.2] 


which  is  solved  using  Newton's  method. 

If,  instead,  the  estimate  of  step  3 is  chosen,  then  step  4 will 
estimate  B and  ax  using  estimates  of  Ay,  M,  a,  y,  z at  the  end  of  step 
3.  The  procedure  is  to  prerotate  (u,v)  by  y and  prerotate  (x,y,z)  by 
a.  Then, it  is  required  to  determine  the  transformation 


u = M(x-Ax)cosB  * z sing 


[20] 


and  the  solution  is  given  by 


— u — 

Ax  = x - - z tanB 

McosB 


[21.1] 


and 
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C cos8  + C sin8  - M(C  - C )sin8  cosB 
uz  ux  v xx  zzJ 

+ MCxz  (sin^B  - cos^B)  = 0 


[21.2] 


Step  5 accepts  the  magnification  and  angle  estimates  and  uodates 
the  translation  and  range  estimates.  After  minifying  the  (u,v) 
coordinates  by  a factor  of  M,  the  transformation  is 


rll  r12  r13 
r2I  r22  r23 


x-Ax  " 

y-Ay 

Z 

J 

_ _ 

[22] 


where  the  matrix  of  r^  is  defined  by  eq.  12.  The  solution  is 


* 

M 

V 


■ 


_ _ 

_ ^ 

-1 

_ — 

Ax 

= 

X 

- 

rll  ri2 

u 

Ay 

y 

. r21  r22  _ 

V 

[23.1] 


rll  r22  ‘ rl2  r21 


r r - r r 
11  13  12  23 


rll  r23  ' r21  r23 
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z = Z + 1 ( r13(u  - u)  + r23(v  - V) 

2 2 ' 
rl3  + r23 

^11  rl3  + r21  r23-* 

(-r!2  r13  + r22  r23i  1 ) 


[23.2] 


Although  the  solution  reauires  soecification  of  only  z,  consistency  with 

the  rest  of  the  algorithm  requires  that  the  specification  of  C be 

zz 

applied  also.  When  this  condition  is  added  to  the  subestimation 
problem,  the  result  is  the  same  except  that  z is  to  be  scaled  to  qive 
the  desired  C 

zz* 

Step  6 performs  an  update  of  ax.  Ay,  M,  y given  a,  b,  z.  The 
solution  is 


r = tan 


-1 


Cuz  sina  + Cuy  COSa  “ Cvx  cosS 


-Cyy  sina  sinB  + Cvz  cosa  sine 


Cvz  Sina  + Cvy  COSa  + cux  cosB 


■♦Cuy  3*na  s*n0  “ Cuz  COSa  s^nB 


[24.1] 
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rll  Cux  + r12  Cuy  + r13  Cuz 

+ r21  Cvx  + r22  ^ + r23  CV2 


S11  Cxx  + s22  Cyy  + s33  Czz 

+ (s12  + s21>  Cxy  + <s13  + S31>  Cxz 
+ (s23  + s32)  Cy2 


Ax 

X 

r, , r, _ 

-1 

u 

— 

11  12 

Ay 

y 

— — i 

r2i  r22 

V 

1 r22 


rll  r22  " rl2  r21 


11 


z 


where  r...  are  as  defined  in  eq.  12  and  are  elements  of  the 
product 

*rll  r21 
r12  r22 
r13  r23 


rll  r12  r13 
r21  r22  r23 


[24.2] 


[24.3] 


matrix 
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In  all  the  applications  of  this  report,  the  algorithm  shows  quick 
convergence.  The  first  iteration  alone  gives  good  results  as 
illustrated  in  Fig.  8 and  Tables  I and  II  using  a pyramid-type 
structure.  Figure  8(a)  shows  a reference  grid  (broken  lines)  associated 
with  the  top  view  of  a pyramid,  on  which  is  superposed  a corresponding 
search  grid  (solid  lines)  where  the  pyramid  has  been  translated, 
magnified,  and  rotated  relative  to  the  reference  position.  The 
transformation  parameters  are  ax  = 1 picture  element.  Ay  = 2 picture 
elements,  M = 1.1,  a = 10“ , 8 = 20“ , y = 30°.  The  magnification  and 
rotations  of  this  example  are  considerably  larger  than  will  be  expected 
to  occur  in  closed  loop  tracking. 

The  remainder  of  Fig.  8 illustrates  the  evolution  of  the 
parameter  estimates  by  transforming  the  reference  grid  at  each  steo 
using  current  estimates  and  superposing  this  on  the  search  grid.  The 
goodness  of  fit  of  the  search  grid  and  the  transformed  reference  grid  is 
a visual  measure  of  the  goodness  of  the  estimates.  Fiqures  8(b)  to  8(f) 
show  how  the  grids  fit  after  each  of  the  first  5 steos  of  the  first 
iteration  of  the  estimation  algorithm.  Note  in  Fig.  8(b)  how  step  1 
has  managed  to  generate  a good  estimate  of  the  roll  angle  y. 

The  results  shown  in  Fig.  8 and  Tables  I and  II  were  obtained 
with  range  statistics  computed  from  exact  values  of  z.  These  statistics 
were  z = 3.2  and  Czz  * 20.48.  Since  these  values  are  generally  not 
known  to  the  tracker,  other  values  were  also  tried,  including  an 
arbitrary  choice  of  z * 0 and  C = i (C  +C ' ) , where  C is  taken 

zz  2 xx  yy  zz 

to  be  the  average  of  C„„  and  c . The  resulting  parameter  estimates 

xx  yy 

were  different,  with  the  exception  of  magnification  and  roll  angle  vrtiose 
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dependence  on  z and  Czz  appears  to  be  only  very  weak.  These  solutions 
were  eouivalent,  however,  both  visually  and  in  terms  of  mean  souare 
error. 


TABLE  I 

Evolution  of  parameter  estimates  and  mean  square  error 
(MSE)  for  pyramid  example 


STEP  # 

AX 

Ay 

Initial 

0.00 

0.00 

Step  1 

2.17 

1.54 

Step  2 

1.05 

1.50 

Step  3 

2.22 

2.73 

Step  4 

1.05 

1.93 

Step  5 

0.96 

1.93 

Step  6 

0.96 

1.89 

Exact 

1.00 

2.00 

1.000 

0.0 

0.0 

0.0 

1.059 

0.0 

0.0 

31.8 

1.085 

0.0 

20.9 

31.8 

1.033 

18.5 

0.0 

31.8 

1.085 

7.5 

20.9 

31.8 

1.085 

7.5 

20.9 

31.8 

1.093 

7.5 

20.9 

30.9 

1.100 

10.0 

20.0 

30.0 

47.02 

1.99 

0.39 

1.51 

0.31 

0.10 

0.06 


■*- — . -Jb 
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TABLE  II 


Initial,  estimated,  and  exact  ranqe  parameters  z for  pyramid  example 


Initial  z 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Estimated  z (after  step  6) 


1.12 

0.97 

0.83 

0.68 

0.53 

0.73 

8.51 

8.36 

8.21 

0.13 

0.33 

8.11 

15.89 

7.81 

-0.27 

-0.07 

7.71 

7.56 

7.41 

-0.67 

-0.47 

-0.62 

-0.77 

-0.92 

-1.07 

Exact  z 
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4.2  Small-Angle  Simulation 


In  the  two-dimensional  case,  it  was  shown  that  the  original 
reference  image  can  he  used  throughout  the  tracking  mission,  the  live 
view  of  the  target  can  always  be  made  to  coincide  with  the  reference 
view  by  simple  magnification  and  roll  angle  adjustment.  Moreover,  these 
correcting  operations  can  be  performed  conveniently  with  a zoom  lens  and 
a gimbal. 

Ibis  is  no  longer  true  when  the  target  rotates  about  all  3 axes. 
Although  the  live  scene  can  theoretically  be  rotated  in  3 axes  given  a 
three-dimensional  description  of  the  taraet,  a sufficiently 
distortion-free  result  would  require  a great  deal  of  depth  information, 
and  hence,  a prohibitive  amount  of  computation  for  a tracking  system. 
Moreover,  the  extent  to  which  such  an  angular  compensation  can  be 
carried  out  is  limited,  since  it  cannot  regenerate  parts  of  the  target 
that  rotate  out  of  view. 

Recourse  to  reference  imaoe  updating  is  therefore  taken. 
Constant  scale  and  roll  attitude  are  still  controlled  with  a zoom  lens 
and  gimbal  combination  to  maintain  as  much  as  possible  the  original 
optimally  chosen  configuration  of  target  size  and  attitude.  Rotations 
about  perpendiculars  to  the  line  of  sight  (angles  a and  (?)  are  estimated 
but  not  compensated  for.  Instead,  a digitally  scale-  and  roll-corrected 
section  of  the  search  scene  is  taken  to  replace  the  old  reference  imaoe. 
The  new  reference,  therefore,  has  the  same  scale  and  roll-orientation  as 
the  old  reference;  it  differs  only  in  the  angles  a and  (?. 
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The  3-D  tracking  algorithm  can  then  be  described  as  reoetition  of 
the  following  sequence: 


1)  Acquire  the  search  scene. 


2)  Determine  the  search  grid  corresponding  to  the  reference  grid 
by  image-matching. 


3)  Perform  a three-dimensional  grid  analysis  as  described  in 
Section  4.1  to  estimate  ax.  Ay,  y and  M. 


4)  Send  ax'.  Ay'  to  sensor  pitch  and  yaw  control. 


5)  Send  M to  200m  lens  control. 


6)  Send  y to  sensor  roll  control. 


7)  Replace  the  reference  image  by  a section  of  search  scene 
digitally  centered  at  (Ax,  Ay),  roll-corrected  by  y,  and 

3 L...  ii 


scaled  by  M, 


! 4 


where,  as  in  the  2-D  case,  (ax'.  Ay')  are  a prerotation  of  (ax  , Ay  ) 
by  the  total  roll  angle  r. 

This  algorithm  was  simulated  on  a series  of  oolaroid  photographs 
of  a military  truck  shown  in  Fig.  9.  After  each  photograph  was  taken, 
the  camera  was  moved  closer  and  to  one  side  of  the  truck  to  produce  an 
approximate  5%  magnification  and  3°  azimuthal  angle  (B)  between 


UNCLASSIFIED 

41 


View  3 


View  5 


FIGURE  9 - Photographs  of  military  truck  used  in  tracking  simulation 
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View  6 View  7 


FIGURE  9 - (continued) 

consecutive  pictures.  The  photographs  were  digitized  to  a resolution  of 

128  x 128.  View  1 was  taken  as  the  first  reference,  and  Views  2 to  7 

served  as  a set  of  six  consecutive  search  scenes.  The  reference  grid 

was  defined  by  a 5 x 8 array  of  points  with  a spacing  of  9 picture 

elements.  Range  parameters  z and  Czz  were  preassigned  values  of  z = 0 

and  C = i (C  + C ) . The  correcting  action  of  the  zoom  lens  was 
2 aa  yy 

simulated  by  preminifying  the  search  scene  prior  to  correlation. 
Similarly,  the  roll  correction  was  simulated  by  prerotation.  Pitch  and 
yaw  corrections  were  not  simulated  since  the  truck  was  already 
sufficiently  centered  in  all  of  the  photographs. 

To  increase  the  speed,  a two-step  approach  was  used  in 
determining  the  search  grid.  First,  a wide  search  for  a 32  x 32  element 
reference  area  using  the  fast  sequential  similarity  detection  algorithm 
provided  the  general  location  of  the  search  grid.  Then,  a set  of 
considerably  narrower  searches  individually  located  the  grid-points.  In 
this  secondary  search,  two  refinements  were  made  for  a more  accurate 
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pinpointinq  of  the  search  qrid  points.  One  of  these  is  the  use  of  the 
cross-correlation  alqorithn  rather  than  the  SSDA.  This  is  because  the 
similarity  function  produced  by  the  hiqhly  nonlinear  SSDA  is  not 
neaninqful  for  interpolation  purposes.  Moreover,  the  local  centroid  of 
the  similarity  function  about  the  peak  is  not  actually  a qood 
interpolation  scheme.  (Amonq  other  problems,  the  centroid  moves  with 
chanqinq  dc  level.)  Hence,  as  a second  refinement,  the  centroid 
calculation  is  replaced  by  maximizinq  a local  second-order  polynomial 
approximation  to  the  cross-correlation  function.  For  simplicity,  2 
one-dimensional  interpolations  alonq  the  x and  y axes  are  made.  The 
coordinates  of  the  interpolated  peak  are  then 
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w*iere  Fw»  Fe'  Fp'  Fs'  Fn  are  nei9hbourinq  function  values  as  shown  below 
and  F is  the  noninterpolated  peak  with  inteqer  coordinates  (x  , y ). 
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For  comparison  purposes,  a parallel  simulation  was  made  to 
illustrate  reference  updatinq  without  benefit  of  qrid  analysis.  In  this 
run,  which  will  be  referred  to  as  1-D  trackinq,  a sinqle  32  x 32 
reference  was  used,  and  only  position  information  (Ax,  Ay)  determined 
the  new  reference.  As  in  3-D  trackinq,  position  was  obtained  by  a wide 
SSDA  search  followed  by  a narrow  cross-correlation  search  with 
second-order  interpolation. 

Parallel  results  of  the  2 simulations  are  shown  in  Fiq.  Id. 
Reference  imaqe  1 is  the  first  reference  and  is  identical  to  View  1. 
Reference  imaqes  2 to  7 are  updated  references  extracted  from  Views  2 to 
7.  The  references  of  the  1-D  simulation  are  shown  on  the  left  side  of 
Fiq.  10  with  the  32  x 32  tarqet  outlined  by  a white  square.  The 
reference  imaqes  of  the  3-D  simulation  are  shown  on  the  riqht  side  of 
Fiq.  10  superposed  with  the  5x5  array  of  reference  qrid  points. 

Inspection  of  Fiq.  10  indicates  that  the  3-D  trackinq  alqorithm 
was  very  successful  in  maintaininq  oriqinal  tarqet  scale  and 
roll-orientation.  Measurements  on  Views  1 and  7 indicate  an  overall 
maqnification  of  1.31.  The  product  of  the  set  of  maanifications 
estimated  by  the  3-D  alqorithm  is  1.311.  Thus  the  3-D  tracker  was  able 
to  effectively  eliminate  a 30%  difference  in  scale  factor  spanninq  the 
series  of  photoqraphs.  For  an  onboard  missile  quidance  application, 
this  feature  of  maintaininq  oriqinal  scale  q^nerally  extends  the  useful 
ranqe  of  the  tracker.  Without  compensation,  imaqe  expansion  durinq 
target  closure  can  lead  to  loss  of  tarqet-defininn  detail,  particularly 
when  the  target  image  overfills  the  field  of  view.  This  can  be  seen  in 
the  references  of  the  1-D  simulation,  where  less  and  less  of  the  truck 
is  included  in  the  target  area.  Apart  from  the  door  marking,  the 
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Reference  image  1 


Reference  image  2 


Reference  image  3 

Reference  updates  generated  during  tracking 
simulation  with  the  truck  target 


FIGURE  10 
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Reference  image  4 


Reference  image  5 


Reference  image  6 


FIGURE  10  - (continued) 
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Reference  image  7 


FIGURE  10  - (continued) 


outlined  target  of  Reference  image  7 retains  very  little  picture  detail 
to  define  a target.  In  Reference  image  7 of  the  3-D  simulation,  on  the 
other  hand,  compensation  for  magnification  resulted  in  retention  of  much 
more  detail  defining  the  cab  of  the  truck  (door,  windshield,  canvas, 
mounting  step) . 

The  other  important  distinction  between  the  2 simulation  runs  is 
in  the  positioning  of  new  references.  In  the  1-D  simulation,  the  system 
essentially  tracked  the  high-contrast  emblem  on  the  door  of  the  truck. 
It  is  evident  in  Fig.  10  that  the  target  updates  followed  the  door 
emblem  (moving  leftward)  as  the  truck  rotated.  Eventually,  if  the 
emblem  were  to  rotate  out  of  view,  the  system  would  momentarily  track 
the  edge  of  the  truck  and  finally  move  into  the  background  with  complete 
loss  of  the  target. 
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In  3-D  tracking,  however,  the  system  determines  a 

three-dimensional  description  of  the  tarqet.  Therefore,  it  has  the 
capability  to  discriminate  between  target  translation  and  target 

rotation  about  perpendiculars  to  the  line  of  sight.  (The  relatively 
unsophisticated  1-D  and  2-D  trackers  both  interpret  rotation  about 

perpendiculars  to  the  line  of  sight  as  translation.)  In  Fig.  10,  the 

resistance  of  the  3-D  tracker  to  move  left  with  the  leftward  rotation  of 
the  truck  is  quite  evident  when  the  grid  position  relative  to  the  door 
emblem  is  noted. 

4.3  Large-Angle  Simulation 

To  test  the  performance  of  the  3-D  tracker  under  large  angles  of 
rotation,  a more  convenient  simulation  was  set  up  usinq  a 
computer-generated  cylinder  as  the  target.  The  cylinder  imagery  was 
generated  by  mapping  segments  of  a multishaded  input  plane  (Fig.  11(a)) 
onto  a senicylindrical  surface,  producing  output  cylinder  images  as 
shown  in  Fig.  11(b).  The  purpose  of  the  random  shading  is  to  provide 
picture  detail  for  tracking.  The  mapping  incorporated  a reflection 
coefficient  consisting  of  the  cosine  of  the  angle  between  the  viewing 
axis  and  the  normal  to  the  surface.  Rotation  about  the  x-axis  (anqle  a) 
was  performed  simply  by  the  appropriate  selection  of  what  segment  of  the 
input  plane  to  map. 

In  this  cylinder  experiment,  exact  knowledge  of  the  neonetry 
together  with  nearly  error-free  search  grids  allowed  a clear  observation 
of  the  tracker's  reaction  to  target  rotation.  It  becane  apparent  that 
exact  tracking  actually  requires  the  use  of  the  exact  values  oF  the 
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ranqe  parameters  z and  Czz.  Althouqh  the  value  of  C zz  has  a weak  effect 
on  tracking  accuracy,  the  value  of  z has  a strong  effect  at  large 
angles.  This  results  from  the  fact  that  the  mean  parameter  z defines 
what  the  tracker  takes  to  be  the  geometric  center  of  the  target.  If 
this  point  is  not  a valid  center  of  rotation  within  the  target,  then  the 
three-dimensional  target  motion  as  perceived  by  the  tracker  will  include 
extraneous  translations,  with  consequent  positional  update  errors.  If  z 
is  made  too  small,  the  system  will  tend  to  undercorrect;  that  is,  a 
reduced  ability  to  maintain  its  aim  on  target  center  and  it  will  have  a 
tendency  to  follow  the  original  reference  imaqe  as  it  rotates  out  of 
view.  If  z is  made  too  large,  the  system  will  overcorrect;  it  will  move 
its  aim  too  far  in  the  direction  opposite  to  the  target  rotation. 

All  this  does  not  mean  that  good  tracking  cannot  be  achieved 
without  exact  knowledge  of  z and  C^.  If  the  target  is  a reasonably 
regularly  shaped  object,  then  approximate  values  of  z and  C zz  that  place 
the  center  of  rotation  almost  anywhere  within  the  target  will  allow  3-D 
tracking  to  a certain  extent.  The  more  the  target  rotates  away  from  the 
original  reference  attitude,  the  more  accurate  an  estimate  of  z is 
required  to  maintain  lock  on  target. 

Based  on  the  preceding  argument,  approximate  values  of  z and  C2z 
are  obtained  by  modelling  the  target  as  a spherical  object.  The  points 
of  the  5x5  reference  grid  are  therefore  modelled  as  points  on  the 
surface  of  a sphere.  Since  the  reference  grid  is  supposed  to  more  or 
less  represent  the  target,  the  modelled  grid  is  made  to  inscribe  the 
outline  of  the  sphere  as  diagrammed  in  Fig.  12.  In  this  experiment,  the 
grid  spacing  is  8 picture,  elements.  Therefore  the  radius  of  the  sphere 
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is  22. P picture  elements  and  the  range  statistics  are  z = 14. ^ and  C„  ^ - 
48.0.  The  corresponding  range  statistics  of  the  same  gri<;  on  the 
cylinder  (vrtiose  radius  is  24  picture  elements)  are  ~z  = 21.0  and  Czz  = 
6.7.  In  this  case,  the  spherical  target  model  leads  to  a significantly 
underestimated  z and  very  overestimated  Czz» 


Outline 
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For  the  cylinder  trackinq  simulation,  a series  of  views  was 
generated  with  increasing  angles  of  ~v] inder  rotation  about  the  x-axis. 
The  relative  angle  of  rotation  between  consecutive  views  was  5°.  As  in 
the  simulation  with  the  series  of  truck  photographs,  both  the  1-D  and 
3-D  trackinq  algorithms  were  applied.  The  only  difference  here  was  the 
use  of  range  parameters  given  by  the  spherical  target  model. 

Results  of  the  parallel  1-D  and  3-D  simulations  are  shown  in  Fiq. 
13  (again  with  1-D  references  on  the  left  and  3-D  references  on  the 
right).  Although  a reference  update  was  made  at  every  5°  interval,  Fiq. 
13  shove  only  the  references  obtained  from  views  corresponding  to  a = 
0°,  30“ , 50“ , 90°  and  120“.  It  is  apparent  that  the  underestimation  of 

z by  the  spherical  target  model  initially  caused  the  3-D  alqorithn  to 
undercorrect  and  tend  to  follow  the  rotation  of  the  cylinder.  As  the 
updated  reference  grid  moved  away  from  center,  however,  the  true  value 
of  z decreased  and  approached  the  value  given  by  the  spherical  target 
model.  While  this  was  happening,  the  upward  motion  of  the  reference 
grid  slowed  down  and  the  system  stabilized  at  a point  on  the  cylinder 
where  the  true  z equals  the  estimated  value  of  z used  by  the  3-D 
algorithm.  This  stabilization  occurred  in  this  case  at  a = 45°.  From 
that  point  on,  the  system's  aim  remained  stable  on  the  cylinder  while 
the  cylinder  was  rotated  further  to  a = 120°. 

The  1-D  tracker,  on  the  other  hand,  tried  to  follow  the 
cylinder's  image  rather  than  track  it  as  a three-dimensional  object. 
Consequently,  its  aimpoint  rolled  off  target  as  the  cylinder  rotated. 
At  a = 60°  the  aimpoint  was  approximately  on  the  edge  of  the  cylinder. 
The  32x32  (updated)  reference  image  was  then  half  cylinder  and  half 
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Reference  image  25 
a = 120° 

FIGURE  13  - (continued) 


background.  At  this  point,  if  the  cylinder  had  been  moving  across  a 
non-empty  background,  the  background  would  quickly  have  taken  over  as 
the  target.  The  tracker  would  then  have  been  left  fixed  on  a point  in 
the  background  and  it  would  have  lost  the  target  entirely.  In  this 
simulation,  the  black  background  did  not  exert  any  pull  on  the  tracker's 
aim,  and  thus  the  aiming  point  moved  relatively  slowly  away  from  the 
cylinder  even  utien  the  reference  image  contained  very  little  of  the 
cylinder.  Figure  14  compares  the  performances  of  the  1-D  and  3-D 
algorithms  in  terms  of  aiming  point  deviation  from  the  center  of  the 
cylinder.  The  stabilization  of  the  3-D  tracker  after  about  45°  of 
cylinder  rotation  is  clearly  evident. 
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Aimpoint  deviation 


from  cylinder  axis 


Edge  of  cylinder 


FIGURE  14 


Aimpoint  deviation  produced  by  1-D  and  3-D  tracking 
algorithm,  as  a function  of  rotation  angle  a 
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Although  the  2-D  tracker  is  a non-updating  algorithm  intended  to 
be  restricted  to  the  two-dimensional  case,  it  is  instructive  to  note  the 
consequence  of  its  application  to  the  three-dimensional  problen;  that 
is,  an  updating  version  of  the  2-D  tracker  applied  to  a target  rotating 
in  three  axes.  In  comparison  to  the  1-D  tracker,  the  2-D  tracker  is 
more  sophisticated  in  that  it  can  detect  and  compensate  for 
magnification  and  rotation  about  the  line  of  sight  (roll).  It  is  not 
capable  of  perceivinq  the  target  as  a three-dimensional  object,  and  it 
is,  therefore,  still  restricted  to  following  images.  Like  the  1-D 
tracker,  its  aim  will  tend  to  roll  off-target  as  the  original  reference 
view  rotates  out  of  sight.  Although  the  2-D  tracker  does  not  detect 
rotation  about  perpendiculars  to  the  line  of  sight,  it  does,  however, 
misinterpret  the  distorting  effect  such  rotation  has  on  search  orids. 
As  an  example,  without  three-dimensional  intelligence,  the  2-D  tracker 
erroneously  interprets  a vertically  squashed  search  grid  from  the 
rotating  cylinder  as  an  overall  reduction  in  the  size  of  the  target.  It 
therefore  instructs  the  zoom  lens  control  to  perform  a compensatory 
magnification.  Thus  the  reaction  of  the  2-D  tracker  to 
three-dimensional  target  rotation  includes  magnifying  ' . target  image. 
This  behavior  is  evident  in  the  cylinder  simulation  in  the  first  step  of 
the  three-dimensional  coordinate  estimation  algorithm  (this  first  step 
is  identical  to  the  two-dimensional  coordinate  estimation) . By  the  time 
the  cylinder  has  rotated  to  120"  in  the  simulation,  the  initial 
two-dimensional  algorithm  interpreted  the  5"  step  in  rotation  about  the 
x-axis  as  a reduction  in  scale  by  a factor  of  3.975.  (This  would  have 
caused  a magnification  by  a factor  of  1.0255.) 
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5.0  APPLICATION  TO  MISSILE  GUIDANCE 


Implementations  of  the  2-D  and  3-D  trackinq  alaorithms  would 
produce  very  different  missile  quidance  systems.  Hie  computational 
workload  of  the  2-D  algorithm  can  be  handled  by  a medium  performance 
microprocessor  supported  by  high-speed  diqital  imaqe-matchinq  hardware. 
The  2-D  algorithm  is  therefore  suitable  for  a missile-borne  trackinq 
system.  This  is  a fire-and- forget  missile  system,  in  vrtrich  the  missile 
is  totally  autonomous  after  launch.  Durinq  target  closure,  the  2-D 
tracker  performs  its  own  magnification  and  roll  anqle  corrections.  Hie 
lack  of  compensation  for  rotation  in  the  other  2 axes  restricts  usage  of 
the  2-D  system  generally  to  fixed  land  targets,  preferably  in  the 
air-to-ground  configuration. 

Like  the  2-D  tracking  algorithm,  the  3-D  alqorithm  controls  scale 
and  roll  angle  (via  zoom  lens  and  roll  gimbal)  according  to  information 
obtained  from  grid  analysis.  It  also  uses  this  information,  however,  to 
update  the  reference  image  by  periodically  takinq  a geometrically 
corrected  search  scene  as  a new  reference.  The  geometric  precorrection 
of  the  search  scene,  based  on  qrid  analysis  data,  consists  of  a 
simultaneous  translation  (horizontal  and  vertical) , change  of  scale,  and 
rotation  (roll  angle).  This  transformation,  involvinq  interpolation  in 
two  dimensions,  is  quite  a massive  task  of  floatinq  point  computation. 
The  required  processing  power  would  be  bulky  and  rather  expensive  for  an 
expendable  system.  Because  of  this,  the  3-D  trackinq  algor ithn  is 
probably  not  feasible  as  a missile-borne  quidance  system. 
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A nonexpendable  application  for  the  3-D  algorithm,  in  which  the 
size  and  weight  restrictions  are  relaxed,  is  an  automatic  airborne  laser 
designator.  In  this  configuration,  the  pilot  defines  the  tarqet,  hands 
off  control  to  the  automatic  designator  and  launches  the  missile.  From 
that  point  on,  the  system  is  autonomous,  leaving  the  pilot  free  to 
maneuver  without  the  burden  of  directing  the  missile.  The  designator 
tracks  it  and  maintains  a laser  beam  on  the  tarqet.  The  task  of  the 
missile  itself  is  simply  to  fly  toward  the  laser  light  reflected  by  the 
target.  This  greatly  reduces  the  cost  of  the  missile  system  since  the 
sophisticated  tracking  system  is  shared  by  a large  nunber  of  relatively 
inexpensive  missiles.  Another  advantage  is  that  the  pilot  can,  at  any 
time,  take  control  to  redefine  the  tarqet  to  correct  a break-lock  or  an 
initial  aiming  error.  The  disadvantage  of  such  a system  is  that 
although  it  is  a short-term  fire-and- forget  system  freeing  the  pilot 
from  the  designation  task,  it  is  not  long-term  fire-and-forqet  in  the 
sense  that  the  pilot  is  not  free  to  escape  outside  the  designator's 
range  until  termination  of  the  missile's  flight. 

Although  3-D  trackinq  is  required  against  maneuvering  tarqets,  it 
is  also  essential  against  fixed  tarqets  when  configured  as  an  airborne 
designator.  This  is  because  a tarqet's  aspect  will  vary  as  the  aircraft 
maneuvers  around  the  target  area.  Consider,  as  a simple  example,  an 
attack  mission  in  vhich  a 700-mi/h  aircraft  in  straight  and  level  flioht 
launches  a 1000-mi/h  missile  toward  a fixed  land  target  at  a 45° 
lookdown  angle.  At  missile  impact,  the  aircraft  would  be  directly  above 
the  target.  This  is  a change  in  taraet  aspect  with  respect  to  the 
aircraft.  Accurate  tracking  and  designation  throughout  missile  flight 
would  therefore  require  3-D  trackinq. 
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Apart  from  noise  and  atmospheric  conditions,  range  of  operation 
is  determined  by  the  optics  and  the  physical  size  of  the  target.  If  the 
format  of  the  imaging  sensor  has  a horizontal  dimension  H,  and  the 
required  horizontal  field  of  view  angle  at  launch  is  e,  then  the 
required  telephoto  focal  length  of  the  zoom  lens  is  given  by 

_2_ 


small  focal  lengths  are  highly  desirable  for  compact  packaging  and 
convenient  gimballing  of  the  optics.  For  a given  field  of  view,  a snail 
focal  lehgth  requires  a anall  image  format.  In  this  application,  the 
small  format  of  charge-coupled  device  (CCD)  imaqinq  sensors  (Ref.  10) 
may  be  used  to  advantage.  As  an  example,  the  image  format  of  a 
commercially  available  512  x 320  element  CCD  imaqer  is  7.30  m x 9.75  m 
(H  = 9.75  nm) . A telephoto  focal  length  of  75  nm  would  qive  a total 
field  of  view  of  about  7.5  degrees.  If  the  tarqet  occupies  32  elements 
of  the  320-element  horizontal  scan,  then  it  would  occupy  a field  of  view 
of  0.75  degree.  This  is  the  anqle  subtended  by  a 50-m  target  at  a range 
of  about  4 kilometres. 

As  the  missile  closes  in  on  tarqet,  the  zoom  lens  must  be 
directed  toward  the  wide-angle  position  (decreasing  focal  length).  Hie 
greater  the  zoom  ratio,  the  closer  the  missile  can  approach  the  target 
before  it  can  no  longer  maintain  original  target  size  in  its  field  of 
view.  With  a zoom  ratio  of  10,  in  the  example  just  described,  the 
missile  could  close  in  to  within  400  m of  the  tarqet  without  losing  the 
original  scale. 


UNCLASSIFIED 

SI 


S.fl  CONCLUSION 


TVra  algorithms  for  determining  relative  inane  magnification  and 
rotation  have  been  presented.  The  information  provided  by  these 
algorithms  can  he  used  by  an  area  correlation  tracker  to  perform  zoom 
and  roll  corrections  so  as  to  maintain  original  target  scale  and  roll 
attitude.  This  allows  a missile  with  an  onboard  area  correlator  to 


approach  a fixed  target  without  updating  the  reference  image,  thus 
avoiding  updating  errors. 


The  algorithms  operate  by  defining  a grid  of  points  on  the 
reference  target  imane  and  determining  the  corresponding  grid  of  points 
in  the  search  scene  by  matching  image  segments.  The  geometric 
relationship  between  the  2 images  (translation,  magnification,  rotation) 
is  given  by  a least  squares  estimate  of  the  coordinate  transformation 
between  the  reference  grid  and  the  search  grid. 


The  2-D  algorithm  assumes  that  the  target,  as  viewed  by  the 
sensor,  does  not  rotate  about  perpendiculars  to  the  line  of  sight.  This 
is  the  case  of  a missile  flyinq  along  the  line  of  siqht  toward  a fixed 
taraet.  In  this  case,  the  coordinate  transformation  between  grids  can 
be  described  in  2 dimensions  and  the  estimate  of  the  transformation  is 
given  in  closed  form.  A simulation  with  an  aerial  infrared  photoqraph 
of  a highway  illustrates  the  2-D  algorithm's  capability. 


When  there  are  rotations  about  the  line  of  sight,  as  in  the  case 
of  maneuvering  targets,  the  geometric  relationship  between  reference  and 
search  grids  must  be  described  in  3 dimensions.  Since  images  are 
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two-dimensional  and  do  not  provide  direct  depth  information,  the  3-D 
algorithm  estimates  not  only  the  three-dimensional  coordinate 
transformation,  but  the  missing  depth  information  as  well.  Mot  all  of 
the  depth  information  is  recoverable,  however,  and  the  solution  is  not 
unique.  A priori  estimates  of  the  mean  and  variance  of  grid  point  depth 
must  he  provided  to  determine  the  solution.  The  mean  defines  for  the 
tracker  the  target's  center  of  rotation,  and  the  variance  gives  the 
tracker  a measure  of  the  overall  depth  of  the  target.  The  solution  is 
given  as  an  alqorithm  of  successive  approximation. 

As  in  the  2-D  algorithm,  the  3-D  tracker  corrects  for 
magnification  and  roll  via  a zoom  lens  and  roll  gimbal.  No  compensatory 
action  can  be  taken,  however,  to  eliminate  effects  of  rotation  about 
perpendiculars  to  the  line  of  sight.  Hie  3-D  tracker  therefore  updates 
the  reference  by  selecting  a scale-and  roll-corrected  search  scene  as  a 
new  reference  image.  Although  the  target  aspect  may  change,  the  scale 
and  roll  attitude  of  the  reference  image  remains  the  sane. 

A tracking  system  equipped  with  3-D  processing  acquires  an 
important  distinction  over  other  area  correlation  systems.  An  area 
correlator  without  grid  processing  is  essentially  a point-tracker;  that 
is,  it  basically  tracks  a single  point  on  the  target.  With  2-D 
processing,  the  system  tracks  a collection  of  target-points  which  are 
assined  to  lie  in  a plane  perpendicular  to  the  line  of  sight.  Such  a 
system  can  maintain  a constant  target  scale  and  roll  attitude  in  its 
field  of  view  and,  thus,  can  be  considered  to  be  trackinq  an  image  of 
the  target  rather  than  simply  a point.  Like  the  point-tracker , however, 
the  image  tracker  wauld  fail  as  the  original  view  of  the  target  rotates 
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out  of  view.  Only  3-D  processing  allows  the  system  to  track  a tarqet  as 
a three-dimensional  object  and,  thus,  to  maintain  aim  at  or  close  to 
target  center  even  after  the  original  view  has  disappeared. 

TWo  simulations  were  performed  to  test  the  3-D  trackinq 
algorithm.  In  a simulation  utilizing  a series  of  photographs  of  a 
military  truck,  the  3-D  algorithm  proved  successful  at  maintaining 
correct  aim  and  constant  target  image  scale  in  the  face  of  magnification 
and  three-dimensional  rotation.  The  resistance  of  the  3-D  tracker's  aim 
point  to  roll  ofttarget  with  target  rotation  was  further  tested  on  a 
series  of  computer-generated  and  computer-rotated  cylinder  imaqes.  The 
aim  remained  close  to  center  for  cylinder  rotation  as  great  as  120 
degrees.  It  was  shown  that  exact  trackinq  with  zero  wander  requires 
exact  knowledge  of  depth  mean  and  variance.  If  the  depth  mean  is 
underestimated,  the  tracker's  aim  will  initially  follow  the  rotation  of 
the  target,  but  will  stabilize,  however,  at  a point  where  actual  depth 
mean  equals  the  assuned  depth  mean. 

The  2-D  trackinq  algorithm,  implemented  in  a microprocessor 
supported  by  image-matching  hardware,  is  suitable  as  the  basis  of  a 
missile-borne  fire-and-forqet  guidance  system.  The  most  appropriate 
application  would  be  air-to-ground  aqainst  fixed  targets.  Tlie  3-D 
algorithm  is  not  likely  to  be  suitable  for  a missile-borne  guidance 
system  because  of  the  heavy  computation  involved  in  the 
geometrical lycorrected  reference  updatinq  scheme.  A more  promising  role 
for  the  3-D  tracker  would  be  an  automatic  airborne  laser  designator. 
Because  of  target  aspect  variations  caused  by  aircraft  flight,  3-D 
tracking  is  advantageous  in  this  application. 
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